1. Field of the Invention
The present invention relates to a method and an apparatus for removing a scratch, an unnecessary element, and noise from a digitized motion picture to repair the motion picture.
2. Description of the Related Art
An old movie film usually has much dust and a lot of scratches. When such a film is electronically read through a scanner and converted to a digital picture, it is necessary to electronically remove dust and scratches. Since what the original image should be cannot be found, pixels at the dust and scratches have no information. It is necessary to repair the image according to the information of surrounding pixels.
In a recent movie, a special effect is employed in many cases in which a scene is taken with an actor or a vehicle flying in the air by the use of wire and the wire is electronically deleted afterwards. Also in such cases, since portions hidden by the wire, which should have been seen, cannot be repaired from the taken film, it is necessary to reproduce them. This can be handled as a case for repairing a pixel which lacks its information.
In addition, for an image in which an error occurs during radio transfer or computer-network transfer, it is required to use a repair technique to make a defect caused by the error inconspicuous.
To retouch an image, the most traditional and the most widely used method is retouching by hands. Software (called paint software) which implements on a computer an operation for drawing a picture on paper with a pen or an air brush is used to remove a scratch in an image by drawing a figure on the image by the use of an electronic pen or an electronic brush.
It is of course a hard job to repair one frame completely. It is further difficult to repair a motion picture. In other words, even the same pixels in frames are not uniformly retouched since hand retouching is used, and as a result, if each frame is acceptably retouched when seen independently, a human eye notices retouching discontinuity in the time domain as noise when the motion picture formed of the frames is continuously seen. Removing this noise is extremely difficult. It is needless to say that this work requires time and skill. It is not rare that retouching a motion picture formed of about 100 frames takes about one week, and this is a large factor to reduce time efficiency in video operations in a movie or a video product.
An attempt has been made to automate image repair with the use of computer image processing in order to make the work efficiently.
A typical case is the use of an image processing filter. With the use of a low-pass filter (low-frequency passing filter), for example, portions which lack information are blurred to fit the surroundings. Repair is possible to some extent. Just with a low-pass filter, the entire image become blurred. To improve this point, many filters have been examined, including a median filter, which uses statistical operations. An image-repair technique using such a filter is described in detail, for example, in xe2x80x9cFundamentals of digital image processingxe2x80x9d written by A. K. Jain, Prentice hall, which is one of main books in image processing. These filters are mainly useful for a lack of information at a small, discrete pixel, but are not suited to repair a large area formed of continuous pixels.
Therefore, these filters are not sufficient for removing a scratch in an image, for repairing the background after an unnecessary element such as wire is removed, or for repair of burst errors during image transfer.
A scratch in a digital image means unnecessary pixels in the image. Scratch noise is formed of a number of adjacent pixels rather than a small independent group of pixels. In an image, scratch noise is formed of, for example, a pixel area of 10 pixels by 100 pixels, and is positioned in an image area having details such as a texture or an edge cut by the scratch. To remove the scratch from the image, the pixels constituting the scratch area need to represent the original data or to be replaced with data equivalent to the original data. To retouch the scratch area effectively, the pixel data needs to generate at the scratch area an image as sharp as those at the surroundings. It is required that a clear edge be maintained to be continuous and a texture generated to replace the wire or the scratch pixels be harmonized with the surrounding texture.
The effect of a conventional technique in digital processing changes according to an image feature generated regularly or at random and the relative size of the feature in the image. A feature generated regularly includes a texture in a brick wall or a cloth. A feature generated at random includes an asphalt road, a concrete road or sandy beach. The effect of the conventional technique also depends on the size and the type of a scratch area. A small, isolated group of noise pixels in a smooth or blurred area in an image can be relatively easily removed by the use of conventional techniques, such as filtering, cloning, and painting. The conventional noise-removing techniques, however, cannot be applied to a scratch formed of a number of adjacent pixels, or a scratch at a texture area or at an area having conspicuous edge or line in an image. The conventional techniques in image noise removal can be classified into two types, (1) intra-frame technique and (2) inter-frame technique. The difference between these two general techniques is mainly where data which substitutes for a noise pixel is obtained. In the inter-frame technique, a pixel required for substituting for a noise pixel is generally copied from a previous frame or continuous frames. On the other hand, in the intra-frame technique, data in an image frame to be repaired is generally used for substituting for a noise pixel.
The inter-frame technique cannot achieve a good result in a picture in which a camera moves extremely or in a scene having a movement. These techniques cannot be applied either to a case in which a scratch extends to several image frames, or to a case in which a damaged image frame is the only one image frame for providing data used for repairing the image. Unfortunately, a scratch generally extends to a number of image frames due to the movement of a movie film in a general projector. In such a case, the corresponding image data obtained from the previous image frame or continuous image frames cannot be used for substituting for a scratch area or for copying. To remove scratch noise, the conventional techniques provide some methods: (a) low-pass filtering and other linear filtering, (b) median filtering and other non-linear filtering, (c) statistical combination of textures, (d) cloning, including copying other parts in an image, (e) painting by hand, (f) method based on projection, and (g) method based on solving a simultaneous equation. With these methods, the best result cannot be obtained in removing various scratches or in removing a scratch in various image conditions.
It is considered that all image signals are formed of a combination at a certain ratio of many sine-wave signals having different frequencies. To analyze such an image as image components, fast Fourier transform (FFT) is used. Processing for the information generated by FFT is performed in the frequency domain. Direct processing for an image signal is performed in the spatial domain.
The methods (a) to (e) are used in either one of the domains. In methods used in the frequency domain, the structure of the entire image can be obtained, but local processing control (line continuity and clearness) is lost. As a result, a line and other details become unclear.
On the other hand, in methods used only in the spatial domain, local processing control and local information are obtained, but information on the structure of the entire image cannot be obtained. A limitation in a local vicinity is caused by a restriction in actual computer processing in some cases. Some methods similar to intermediate filtering are not capable of using the entire information effectively.
In a conventional image repair method, areas used for repair are obtained from the previous and next frames by copying. As shown in an example illustrated in FIG. 1, the i-th frame 31 shown in B includes a scratch (thick, black portion) 3.
In the figure, the (ixe2x88x921)-th frame 30 illustrated in A, the i-th frame 31 illustrated in B, and the (i+1)-th frame 32, a moving object 41 moving toward the right bottom is shown at almost the center. At almost the center of the i-th frame 31 shown in B, the almost rectangular scratch 3 extending to almost the right top is included. In the (ixe2x88x921)-th frame 30 illustrated in A and the (i+1)-th frame 32 illustrated in C, portions 7 having the same shape as the scratch 3, which is included in the i-th frame 31 shown in B, are set at the corresponding positions. Data under the scratch 3 in the i-th frame 31 illustrated in B can be repaired by copying an image area obtained from the portions 7, which are shown by dotted lines in the previous and next frames.
Brightness and shade levels may change among frames in some cases as shown in FIG. 2.
In the figure, the (ixe2x88x921)-th frame 30 illustrated in A, the i-th frame 31 illustrated in B, and the (i+1)-th frame 32 illustrated in C, a moving object 41 having the same shape as the moving object 41 shown in FIG. 1 is shown. Each of the frames 30 to 32 has different brightness and shade level. As compared with the moving object 41 shown in the i-th frame 31, the moving object 41 shown in the (ixe2x88x921)-th frame 30 is darker, and the moving object 41 shown in the (i+1)-th frame 32 is brighter. In such a case, color does not match if an area is just copied and pasted. Repair quality is low.
In some cases, a scratch 3 largely overlap among adjacent frames as shown in FIG. 3.
In the figure, the (ixe2x88x921)-th frame 30 illustrated in A, the i-th frame 31 illustrated in B, and the (i+1)-th frame 32 illustrated in C, a moving object 41 having the same shape as the moving object 41 shown in FIG. 1 is shown. At slightly left of the center of the (ixe2x88x921)-th frame 30 shown in A, a rectangular scratch 3 extending to almost the upper right is included. At almost the center of the i-th frame 31 shown in B, the rectangular scratch 3, extending to almost the upper right, is included. At slightly right of the center of the (i+1)-th frame 32 shown in C, the rectangular scratch 3, extending to almost the upper right, is included. The scratch 3 largely overlaps among the adjacent frames.
In such a case, it is necessary to obtain data for repair from a frame further apart in the time domain, such as the (ixe2x88x922)-th frame or the (i+2)-th frame. When a frame further apart in time is referenced, since a difference in a shade level becomes larger, it is difficult to perform a good repair.
As described above, in the conventional techniques, it takes much time to repair a motion picture by hand even by a skilled operator. Even with this work, it is difficult to perform uniform repair over all frames. As a result, repair quality is low.
The conventional computer-repair method does not have a sufficient effect against lack of information at a number of continuous pixels. The method cannot be used for continuous, uniform repair against a motion picture. Even with the technology disclosed in Japanese Patent Application Nos. 7-303420 and 8-204776, which has greatly improved repair precision against a single frame, quality is not good when a motion picture is repaired.
In a method in which repair is performed by copying an image from the previous and next images, when brightness and a shade level change as time elapses, a good result cannot be obtained. If a portion lacking information overlap among adjacent frames, a frame further apart needs to be used. When a frame further apart in time is referenced, however, since a difference in a shade level or a shape becomes larger, it is difficult to repair an image.
As described above, with the conventional methods, a single frame may be repaired with a good result, but a good repair for the entire motion picture is impossible.
Accordingly, it is an object of the present invention to provide a motion-picture repair method and a motion-picture repair apparatus which allow even a motion picture to be repaired successfully.
The foregoing object is achieved according to one aspect of the present invention through the provision of a motion-picture repair method in which a pixel which lacks information is repaired in a motion picture formed of a plurality of continuous images, including: a continuity maintaining step for repairing the pixel which lacks information such that the images are continuous in the time domain, according to the information of the image area corresponding to the pixel which lacks information, in at least one image of the previous and next images of the image to be repaired having the pixel which lacks information.
The foregoing object is achieved according to another aspect of the present invention through the provision of a motion-picture repair apparatus in which a pixel which lacks information is repaired in a motion picture formed of a plurality of continuous images, including: control means for controlling continuity maintaining processing to repair the pixel which lacks information such that the images are continuous in the time domain, according to the information of the image area corresponding to the pixel which lacks information, in at least one image of the previous and next images of the image to be repaired having the pixel which lacks information.
The foregoing object is achieved according to still another aspect of the present invention through the provision of a medium for providing control information to have a control apparatus execute repair processing for repairing a pixel which lacks information in a motion picture formed of a plurality of continuous images, the medium including control information for repairing the pixel which lacks information such that the images are continuous in the time domain, according to the information of the image area corresponding to the pixel which lacks information, in at least one image of the previous and next images of the image to be repaired having the pixel which lacks information.